chore(#10700): re-write Enketo form save workflow by jkuester · Pull Request #11256 · medic/cht-core

jkuester · 2026-07-17T01:50:59Z

Description

This PR basically re-writes the doc-saving workflow when an Enekto form (either contact/app/training form) is completed. The existing legacy code suffered from several major problems that made it very difficult to continue developing features on top of it:

The doc-processing pipeline for contact docs vs reports was almost completely disjointed. Aside from the core xml-to-object functions, none of the other logic/workflows was coherently shared between contacts and reports. This meant that we had a lot of duplication and inconsistency between how things were being handled. And, any new features (e.g. attachments) needed to be essentially implemented twice.
Historical changes to these workflows have been haphazard. The current structure is now incredibly complex with very poor abstraction/encapsulation. It is difficult to understand the data flow and/or which parts of the code are responsible for what.
The code does a poor job of communicating the baked-in assumptions (instead nesting the handling of them deep in the workflows).

The main impetus behind is PR is to provide a better structured base for solving #10700. The hope is that the new functionality in #10923 can be ported on top of the changes in this PR in a way that is way more simple and maintainable.

What is in this PR

The goal was to have this PR be a complete drop-in replacement for the existing functionality. I have done my best to ensure the new code accurately matches the behavior of the old code and there are no new features that I intended to directly support by these changes.

The only behavior difference that I am aware of currently is a new error I added to the save workflow that will fail with a helpful error message when a contact form does not have the contact data in a group that is named for the actual contact type. (So, a person-create form needs to have the contact data in the person group.) This requirement is listed explicitly in our docs and it is implicitly expected by the pre-population logic for setting the initial data onto the contact. Previously it was technically possible to save a contact form with data in a different group. Because our documentation is explicit about how the forms are supposed to be structured, I do not see this as a breaking change. Also, if some does have this issue, they will hit the error and be unable to save those contact forms. It will not just cause weird data issues by failing silently. The fix would just be to update the form config to name the group properly.

Added `FormConfig`

A new FormConfig class has been added in webapp/src/ts/services/form/form-config.ts that serves as a wrapper for the form doc and its relevant attachments. This FormConfig is retrieved via the XmlFormsService when rendering the form and then can be re-used when saving the form. This wrapper basically centralizes the handling of the form doc and its attachments and avoids the need to retrieve these things multiple times during the render/save process.

Added new `EnketoFormData` classes

The new EnketoFormData classes added in webapp/src/ts/services/form/form-data.ts are wrappers to encapsulate the XML data model produced when saving an Enekto form. These data classes own the logic for interacting/querying the XML data as well as de-serializing that data to JS objects.

Being able to operate against the XML data is useful for queries (either via css selectors or XPaths) which do not have a simple alternative when using normal JS objects. These wrappers help define the available data queries as well as hold on to the _id references for the data to allow for easy interlinking.

The xml de-serialization logic from the now-deleted EnketoTranslationService was moved into these EnketoFormData classes. (The JS -> XML serialization logic was just moved to where it was being used in the EnketoPrepoulationDataService.)

Move Contact Save functionality into EnketoService

Previously, we had the logic for saving contact forms separated out into the ContactSaveService. I have move this into the EnektoService so now there are only two entry points for saving a form: EnketoService.saveContact and EnketoService.saveReport. Keeping the code in the same service allows for better alignment between the workflows.

As before, there is still probably "too much" code in the EnektoService, but I have balanced things out a bit by deleting duplicate logic and/or moving it into the EnektoFormData classes. The end result is the actual LOC in the enketo.service file has only grown around 30 lines...

Clean up form/enketo service unit tests

A large percentage of the changed lines in this PR are a result of me overhauling the unit tests for the FormService and EnketoService. Previously there was a TON of duplication/leaking between those unit tests (because of how I previously handled separating the FormService from enketo... 😓). The time has come to pay the piper and I have refactored the FormService tests especially to just mock the EnektoService.

AI Disclosure

Basically every single line of the implementation code was hand-written by me. I leveraged Claude a lot for the investigation and minor code structure advice, but ultimately, I really wanted full control/responsibility for porting all the logic line-by-line. Any bugs will be ones that I personally wrote. 👍 😅

I did leverage Claude heavily for the updates to the unit tests. Even there, I have gone through everything line-by-line and made many editorial changes and requested additional tests, etc. There is a lot of lines changed, so it is possible we missed some stuff, but I am highly confident the functionality is well tested and the tests are very maintainable going forwards.

Code review checklist

UI/UX backwards compatible: Test it works for the new design (enabled by default). And test it works in the old design, enable can_view_old_navigation permission to see the old design. Test it has appropriate design for RTL languages.
Readable: Concise, well named, follows the style guide
Tested: Unit and/or e2e where appropriate
Backwards compatible: Works with existing data and configuration or includes a migration. Any breaking changes documented in the release notes.
AI disclosure: Please disclose use of AI per the guidelines.

Compose URLs

If Build CI hasn't passed, these may 404:

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

mrjones-plip · 2026-07-22T18:37:01Z

I came here to see what this was all about and saw this:

I do not see this as a breaking change

Amazing work Josh!!

dianabarsan

Very nice rewrite! I think you've definitely taken this into the right direction.
I left some minor comments and possibly found some issues, but nothing major.

dianabarsan · 2026-07-22T11:50:12Z

+        .map(fieldName => ({ fieldName, doc: formData.getSiblingData(fieldName)?.deserializeDoc(config) }))
+        .map(({ fieldName, doc }) => ({ fieldName, doc: this.initializeContactSibling(rootOutputDoc, doc)}));


Do we need two chained maps here or can everything be done within one iteration?

Wondering if we did something like:

const siblingSave = EnektoContactFormData.SIBLING_FIELD_NAMES.map(async (fieldName) => { const siblingData = formData.getSiblingData(fieldName)?.deserializeDoc(config); const siblingDoc = this.initializeContactSibling(rootOutputDoc, siblingData); rootOutputDoc[fieldName] = await this.getContactSiblingValue( siblingDoc, rootOutputDoc[fieldName], defaultData[fieldName] ); }); await Promise.all(siblingSave);

dianabarsan · 2026-07-22T11:51:10Z

+        _attachments: formAttachments
+      };
+
+      const siblings = EnektoContactFormData.SIBLING_FIELD_NAMES


Please consider moving this sibling handling section into its own function.

dianabarsan · 2026-07-22T11:55:11Z

+    return await this.getContactFromDatasource(Qualifier.byUuid(currentValue._id));
+  }
+
+  private dehydrateContactLineage(contactDoc: Record<string, any>) {


I believe the internal name was "minify", instead of "dehydrate". I'm fine with either, but prefer to have a naming convention.

dianabarsan · 2026-07-27T04:52:18Z

+    const newFileAttachments = FileManager
+      .getCurrentFiles()
+      .map(file => ({
+        name: `${this.USER_FILE_ATTACHMENT_PREFIX}${file.name}`,


Old code stripped special characters (UUID fallback for e.g. Devanagari-only names) and rewrote the referencing field values. The specs checking this were dropped without replacement.

dianabarsan · 2026-07-27T05:30:29Z


-  get formDoc() {
-    return { _id: this.formId };
+  get formConfig() {


Every time we use this getter, it ends up parsing the xml (via the FormConfig constructor). Should we consider cashing here?

dianabarsan · 2026-07-27T06:14:49Z

-    return { _id: this.formId };
+  get formConfig() {
+    return new FormConfig(
+      { _id: this.formId },


Do we need additional fields on this doc here?
saveReport destructures internalId/xmlVersion from config.doc — with only _id, the emitted report docs seem to lose their form field (on master completeNewReport set form: formId), and binary attachments get named user-file/undefined/...

jkuester added 29 commits July 12, 2026 11:20

Initilize NewEnketoService

6df4caf

try to work in contextElement

426cc44

Initial complete

df13baf

Creating new reports working

2cefd19

Edit reports seems to also work

04dc956

Add support for contacts

8aac455

Fix some bugs

e2c6fad

Narrow the CI checks for now

c8572c8

Update service to remove legacy content.

ce42f0f

Avoid writing empty attachments object

56e5eb0

Fix attachment handling on edits

ac02e3d

Update whole flow to use FormConfig

eeeb217

Fix minor issues

b7c6cb8

Fix issue with form validation

b009775

Fix contact type issue

171a25d

Diable transpiling webapp tests

e533acd

Fix FormConfig import to not leak into cht-form

ed3024d

Final clean up for NewEnketoService

bf5286e

Slot everything into place and remove old code.

ec882d7

Fix the cht-form build

6f0c8cd

Fix the sonar issues

d354e56

Lock in the cht-form

e4b7e32

Lock in module unit tests

9e5fa1c

Lock in more unit tests

f10f949

Lock in prepop tests

046178a

Lock in form tests

8975432

Lock in enekto tests

2fd0db6

First round of changes to form-data unit tests

e0e4e72

Lock in form-data

cd23e70

jkuester changed the title ~~Initilize NewEnketoService~~ chore: re-write Enketo form save workflow Jul 18, 2026

jkuester added 2 commits July 20, 2026 10:38

Fix sonar

d09a4db

Update cht-form to clear undefined properties

6844aaf

jkuester commented Jul 20, 2026

View reviewed changes

Comment thread tests/integration/cht-form/default/person-edit.wdio-spec.js

jkuester commented Jul 20, 2026

View reviewed changes

Comment thread webapp/web-components/cht-form/src/app.component.ts

jkuester commented Jul 20, 2026

View reviewed changes

Comment thread webapp/src/ts/modules/contacts/contacts-edit.component.ts

jkuester commented Jul 20, 2026

View reviewed changes

Comment thread webapp/src/ts/modules/contacts/contacts-edit.component.ts

Update cht-form unit tests

f9ab731

jkuester requested a review from dianabarsan July 20, 2026 18:49

Final cleanup

e5cdabc

jkuester marked this pull request as ready for review July 20, 2026 18:50

jkuester changed the title ~~chore: re-write Enketo form save workflow~~ chore(#10700): re-write Enketo form save workflow Jul 20, 2026

jkuester mentioned this pull request Jul 20, 2026

convert-contact-forms: /data/contact group is injected into the wrong group when init contains nested groups medic/cht-conf#832

Open

dianabarsan requested changes Jul 27, 2026

View reviewed changes

jkuester added 3 commits July 27, 2026 12:33

Minor refactors from code review.

dd3c717

Fix findNodeWithTextContent to support special chars

9a88818

Replace getNodeByXpath with new findReferencedDoc logic in EnketoService

c513f16

		.map(fieldName => ({ fieldName, doc: formData.getSiblingData(fieldName)?.deserializeDoc(config) }))
		.map(({ fieldName, doc }) => ({ fieldName, doc: this.initializeContactSibling(rootOutputDoc, doc)}));

Uh oh!

Uh oh!

Conversation

jkuester commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What is in this PR

Added FormConfig

Added new EnketoFormData classes

Move Contact Save functionality into EnketoService

Clean up form/enketo service unit tests

AI Disclosure

Code review checklist

Compose URLs

License

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mrjones-plip commented Jul 22, 2026

Uh oh!

dianabarsan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dianabarsan Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

dianabarsan Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

dianabarsan Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

dianabarsan Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dianabarsan Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

dianabarsan Jul 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jkuester commented Jul 17, 2026 •

edited

Loading

Added `FormConfig`

Added new `EnketoFormData` classes